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OBJECTIVES 

The  objective  of  this  project  is  to  collaborate  with  the  Naval  Research  Laboratory, 
Marine  Meteorology  Division  in  Monterey,  CA  to  develop  novel  and  efficient  data 
compression  techniques  for  weather  data,  with  the  goal  of  meeting  the  challenges  of 
efficiently  storing  and  transmitting  ever-increasingly  large  amounts  of  weather  related 
data. 

APPROACH 

A  meteorological  radar  data  assimilation  system  has  been  developed  at  the  Marine 
Meteorology  Division  of  the  Naval  Research  Laboratory  (NRL)  to  provide  environmental 
information  to  enhance  the  safety  of  ship  and  aircraft  operations.  Radar  observations  are 
assimilated  into  the  Navy’s  Coupled  Ocean/Atmosphere  Mesoscale  Prediction  System 
(COAMPS®)  to  improve  the  forecasts  of  hazardous  weather  and  to  provide  decision 
makers  with  timely  products  to  help  exploit  or  mitigate  those  predictions.  The  system 
will  take  advantage  of  a  selected  group  of  Navy  ships  that  are  to  have  weather  processors 
installed  for  their  tactical  radars  (e.g.,  SPS-48E/G:  Hazardous  Weather  Detection  and 
Display  Capability  (HWDDC);  SPY-1  Tactical  Environmental  Processor).  This  group  of 
ships  will  be  able  to  digitally  generate  full-resolution,  volumetric  weather  radar  data 
every  minute,  and  also  archive  them  to  Universal  Format  (UF)  files.  There  are  plans  to 
transmit  three  UF  files  per  hour  in  near-real-time  to  Fleet  Numerical  Meteorology  and 
Oceanography  Center  (FNMOC)  where  the  data  assimilation  into  COAMPS8  is 
conducted. 

UF  file  sizes  range  from  ~5  MB  (SPS-48E)  to  ~13  MB  (SPY-1),  which  would  be  too 
large  a  load  on  the  operational  bandwidth  of  the  ships’  communication  systems.  To 
overcome  this  obstacle,  we  have  developed  a  novel  UF  file  compressor  that  typically 
reduces  UF  file  sizes  by  a  factor  of  forty,  thus  permitting  their  transmission  from  a  ship  to 
FNMOC. 

The  current  UF  file  compressor  achieved  large  compression  by  exploiting  only  intra-UF 
file  correlations.  Since  UF  files  were  taken  approximately  5  minutes  apart,  temporal 
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correlations  are  expected  to  exist  between  radar  data  in  neighboring  UF  files.  In  this 
work,  we  considered  differential  compression  techniques,  which  aim  at  efficient  UF  file 
transfer  over  a  bandwidth-constrained  link  in  the  case  where  the  receiver  already  has  a 
previous  UF  file  that  has  been  perfectly  reconstructed.  We  employed  the  block-based 
motion  estimation  technique  to  estimate  the  temporal  correlations  between  successive  UF 
files.  Strictly  lossless  coding  was  accomplished  by  compressing  the  motion  parameters 
and  the  residue  signals  after  motion  compensation  using  lossless  coding  methods.  In 
addition,  we  introduced  a  new  fast  block-matching  method  for  lossless  motion  estimation 
in  order  to  significantly  reduce  the  computational  complexity  of  the  inter-file 
compression  method. 

On  the  other  hand,  we  attempted  to  further  improve  the  compression  efficiency  of  the 
current  UF  file  compressor  by  modifying  the  current  low-complexity,  intra-file 
compression  methods,  where  thresholding  operations  were  employed  to  blank  out  data 
entries  whose  values  are  below  certain  thresholds  and  thus  considered  less  critical,  as 
determined  by  some  appropriate  quality  control  requirements.  While  the  current  UF  file 
compressor  achieved  a  high  efficiency,  it  did  not  directly  exploit  these  markers  that 
comprise  a  significantly  large  proportion  of  the  data  to  be  compressed.  To  this  end,  we 
proposed  a  new  data  compression  method,  which  compresses  the  blankout  markers 
separately  from  the  critical  data  entries  in  the  UF  data  files  after  thresholding  operations. 

Besides  studying  radar  data  compression  algorithms,  we  also  investigated  efficient 
further  compression  of  weather  satellite  images  stored  in  JPEG  fonnat.  The  satellite 
images  used  in  the  study  were  downloaded  from  the  NRL,  Monterey  website  for  satellite 
meteorology.  The  images  have  already  been  compressed  using  the  JPEG  standard. 
Further  lossy  compression  of  JPEG  image  in  order  to  reduce  the  size  may  result  in  a  poor 
quality  image.  Thus,  we  studied  the  feasibility  of  losslessly  re-compressing  JPEG  images 
without  loss  of  quality.  To  this  end,  we  tested  and  compared  two  open-sourced  software 
packages:  one  is  PackJPG,  based  on  the  idea  of  segmented  entropy  coding  in  reference  to 
the  EOBs  (End-of-Block)  of  the  8x8  DCT  transformed  macroblocks.  The  other  is  Paq81, 
a  general-purpose  lossless  data  compressor. 


WORK  COMPLETED 

1 .  Improvement  of  the  UF  file  compressor. 

By  collaborating  with  Dr.  Paul  Harasti  at  NRL,  we  improved  the  UF  file  compressor 
UFZIP,  so  that  the  software  could  be  generalized  for  all  UF  file  data  field  identifier 
variations  that  have  changed  in  the  past  between  the  DoD  radars,  SPS-48E,  SPY-l/TEP 
and  SWR.  We  also  modified  the  UF  file  compressor  software  to  determine  several 
parameters  (e.g.,  the  maximum  ray-record  length  size)  dynamically  at  run  time,  rather 
than  specified  the  values  for  these  parameters  statically  in  the  executing  script  as  was 
previously  done. 
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2.  Lossless  differential  compression  of  weather  radar  data  using  motion  estimation  and 
compensation 

In  order  to  assess  the  effectiveness  of  the  inter-file  coding  approach,  we  experimented 
with  two  methods  of  extracting  the  update  infonnation  between  data  from  two  temporally 
adjacent  UF  files,  (i)  Simple  subtraction  of  the  reference  data  from  the  current  data;  (ii)  A 
more  sophisticated  method  known  as  delta  compressor.  We  found  that  neither  of  these 
two  methods  could  provide  more  efficient  compression  than  intra-only  compression, 
which  is  based  on  bzip2.  Thus  we  investigated  using  motion  estimation  techniques  that 
have  been  employed  by  many  video  data  compression  standards  such  as  MPEG  to 
capture  the  temporal  correlations  between  radar  data  from  successive  UF  files,  and  to 
compensate  for  the  ship  movement  and  other  motion  of  weather  prior  to  applying 
differential  coding  codes.  We  analyzed  an  archived  SPS-48E  UF  data  set  provided  by 
NRL  Monterey.  The  data  were  obtained  from  an  at-sea  experiment  onboard  the  USS 
PELELIU  (LHA5)  in  February  2006.  This  data  set  contains  a  wide  range  of  precipitation 
echoes  spanning  22  hours  of  observations. 

3 .  Computationally  efficient  block-based  motion  estimation  algorithms  using  multilevel 
successive  elimination  with  multiple  passes 

Block-based  motion  estimation  was  found  to  be  effective  in  capturing  the  temporal 
correlations  between  weather  radar  data  from  temporally  neighboring  UF  files.  While  the 
exhaustive  full  search  block  matching  method  can  find  globally  optimal  motion  vectors 
without  any  loss  in  the  matching  accuracy,  high  computational  complexity  of  this  method 
limits  its  practical  applications.  Lossless  block  matching  algorithms  based  on  successive 
candidate  elimination  were  found  to  be  effective  in  reducing  the  complexity  of  the  full 
search  method.  We  introduced  a  new  breadth-first  method  with  multiple  passes  to  speed 
up  a  well-known  fast  method  called  multi-level  successive  elimination  algorithm 
(MSEA),  by  exploiting  the  correlations  between  matching  metrics  of  different  levels. 

4.  More  efficient  compression  of  UF  data  by  exploiting  blankout  markers  due  to 
thresholding  operations 

We  introduced  a  new  UF  data  compression  method,  where  we  divided  the  thresholded 
data  entries  into  two  categories:  blankout  markers  and  critical  data  entries.  We 
compressed  these  two  types  of  data  entries  separately  at  the  encoder.  In  order  to  store  or 
transmit  the  location  information  of  the  markers  efficiently,  we  proposed  a  lossless 
method  based  on  an  approach  generally  known  as  DPCM  (Differential  Pulse  Code 
Modulation),  which  compresses  the  location  information  and  perfectly  reconstructs  the 
location  infonnation.  We  applied  the  proposed  method  on  the  UF  radar  data  set  and 
compared  its  compression  efficiencies  with  the  original  method  used  by  the  UF  file 
compressor,  which  directly  compressed  the  thresholded  data  without  separating  the 
blankout  markers  from  the  critical  DZ  data.  We  demonstrated  that  the  new  method 
achieved  higher  compression  than  the  original  method. 
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5.  Further  compression  of  weather  satellite  images  in  JPEG  format 

Since  the  satellite  images  on  the  NRL  web  page  have  already  been  compressed  using  the 
JPEG  standard,  we  can  only  reconstruct  the  images  by  decoding  the  downloaded  JPEG 
bitstreams  that  correspond  to  the  displayed  images  on  the  web.  The  reconstructed  images 
might  be  a  distorted  version  of  the  original  raw  image  data,  due  to  the  lossy  nature  of  the 
JPEG  technique.  In  order  to  assess  the  feasibility  of  further  compressing  the  images,  we 
applied  PackJPG  and  Paq81  on  the  reconstructed  images  and  compare  the  compressed  file 
sizes  with  the  size  of  the  JPEG  bitstreams. 

RESULTS 

1.  Compression  of  UF  data  files  with  the  improved  and  generalized  UFZIP  software 

The  UF  file  compression  software  UFZIP  was  improved  in  terms  of  compression 
efficiencies.  In  addition,  it  was  standardized  to  extend  its  applications  to  the  UF  data  files 
from  other  DoD  Doppler  radars  including  SWR  and  TEP.  Table  1  shows  some  test  results 
with  SPS-48E,  SWRUF  and  TEP  data  files. 

Table  1.  Test  results  of  the  improved  and  generalized  UFZIP  software 


UF  File  Type 

SPS-48E 

SWR 

TEP 

Original  File  Size 

5.4  Mb 

17.0  Mb 

12.1  Mb 

File  Size  After  Compression 

0.245  Mb 

0.34  Mb 

0.13  Mb 

Size  Reduction  Ratio 

22:1 

50:1 

93:1 

2.  Efficient  lossless  differential  compression  of  weather  radar  data  using  motion 
estimation  and  compensation 

Figure  1  (b)  gives  an  example  of  motion  vectors  as  a  result  of  motion  estimation  on  data 
from  two  temporally  neighboring  UF  files  in  Figure  1  (a). 

Figure  1.  Motion  estimation  and  motion  vectors 


First  Tilt  of  DZ  Data  (05:24:39  UTC) 


50  100  150  200  250  300  350 

Azimuth 

First  Tilt  ot  DZ  Data  <05:29:37  UTC) 


(a)  (b) 

(a)  DZ  data  from  two  adjacent  UF  files.  (b)  Motion  vectors 
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As  can  be  seen  from  Figure  2,  the  inter-file  compression  method  outperforms  the  intra¬ 
file  compression  method  consistently  for  all  the  UF  files  in  the  dataset,  thereby 
demonstrating  the  effectiveness  of  the  proposed  method  based  on  motion  estimation  and 
compensation 

Figure  2.  Comparison  of  the  sizes  of  the  compressed  files  between  the  inter-file 
compression  method  based  on  motion  estimation  and  compensation,  and  the  intra-file 
compression  method  based  on  bzip2. 


The  proposed  inter-file  differential  compression  method  is  completely  lossless  and  more 
than  meets  the  1  MB  maximum  file  size  requirement  imposed  by  the  operational 
bandwidth.  However,  in  terms  of  overall  file  size  reduction,  it  cannot  compete  with  the 
existing  intra-file  data  reduction  and  compression  technique  used  in  UFZIP,  which  is 
essentially  a  lossy  compressor.  While  providing  a  useful  lossless  data  compression  option 
for  solving  the  current  problem  of  efficient  file  transfer,  the  proposed  differential 
compression  technique  may  also  benefit  applications  in  efficient  data  archiving,  where 
strictly  lossless  data  compression  is  often  a  requirement. 

3 .  Computationally  efficient  block-based  motion  estimation  algorithm 

We  proposed  a  novel  multiple-pass  method  to  prioritize  the  search  of  different  candidate 
blocks.  By  exploiting  the  inherent  correlations  between  matching  metrics  at  different 
levels,  the  proposed  method  was  shown  to  achieve  an  average  speedup  of  over  1 7  times 
over  the  exhaustive  full  search  method  on  several  benchmark  video  sequences  in  C1F 
format,  representing  a  significant  improvement  over  the  conventional  MSEA  method, 
which  is  known  for  its  high  computational  efficiency  for  block  matching  in  motion 
estimation.  The  proposed  fast  block-matching  method  is  expected  to  be  able  to 
substantially  reduce  the  computational  complexity  of  the  encoder,  if  it  is  applied  to 
compression  of  weather  radar  data, 
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4.  More  efficient  compression  of  UF  data  by  exploiting  blankout  markers 

We  applied  the  a  new  method  of  separately  coding  blankout  markers  and  critical  data 
entries  on  the  UF  radar  data  set,  and  compared  its  compression  efficiencies  with  the 
original  method  used  in  UFZIP,  which  directly  compressed  the  thresholded  data  without 
separating  the  blankout  markers  from  the  critical  DZ  data.  As  can  be  seen  in  Figure  3,  on 
average,  the  new  method  achieves  an  improvement  of  approximately  3%  over  the 
original  method  for  all  UF  files  in  the  data  set,  with  a  maximal  improvement  of  over 
13%. 

Figure  3.  Comparison  of  the  sizes  of  the  compressed  DZ  data. 


5.  Further  compression  of  weather  satellite  images  in  JPEG  format 

Table  2  shows  some  of  test  results  on  satellite  images  taken  for  each  Pacific  area. 


Table  2.  Further  compression  of  satellite  images  in  JPEG  format 


Image  Name 

JPEG 

(Kb) 

PackJPG 

(Kb) 

Paq81 

(Kb) 

20101 120. 0000. goesl  l.vis.fd.x.jpg 

133 

111 

112 

20101 120. 0300. goesl  l.vis.fd.x.jpg 

114 

94.4 

94.5 

20101 120. 1200. goesl  l.vis.fd.x.jpg 

107 

87.5 

87.4 

20101 120. 1500. goesl  l.vis.fd.x.jpg 

124 

103 

103 

20101 120. 1800. goesl  l.vis.fd.x.jpg 

148 

124 

125 

20101 121. 0000. goesl  l.vis.fd.x.jpg 

133 

111 

112 

20101 121. 0300. goesl  l.vis.fd.x.jpg 

114 

94.5 

94.7 

20101 121. 1200. goesl  l.vis.fd.x.jpg 

107 

87.4 

87.4 

Total 

980 

813 

816 
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It  can  be  seen  that  both  PackJPG  and  Paq81  could  achieve  a  further  reduction  of  the 
JPEG  image  sizes  by  about  17%.  In  terms  of  computational  efficiency,  PackJPG  ran 
faster  than  Paq8 1 .  Similar  results  were  obtained  for  a  large  number  of  satellite  images  for 
other  regions.  Thus,  we  demonstrated  it  was  feasible  to  further  compress  the  satellite 
images  stored  in  JPEG  format. 


IMPACT/APPLICATIONS 

Our  attempt  to  apply  motion  estimation/compensation  methods  to  address  the  problem  of 
lossless  compression  of  weather  radar  data  could  be  the  first  of  its  kind,  to  the  best  of  our 
knowledge. 

NRL  has  delivered  the  UF  file  compressor  to  Basic  Commerce  and  Industries, 
Moorestown,  NJ,  who  is  the  developer  of  the  HWDDC  under  contract  for  Space  and 
Naval  Warfare  Systems  Command  (SPAWAR)  Systems  Center,  Pacific,  San  Diego,  CA. 
From  May  to  August  2009,  BCI  and  SPAWAR  conducted  tests  using  the  land-based 
SPS-48E  at  Navy  facilities  in  Dam  Neck,  VA  on  the  HWDDC.  One  of  these  tests  was  the 
successful  transfer  of  compressed  UF  files  from  the  HWDDC  to  NRF  in  real-time.  In 
June  2010,  SPAWAR  Systems  Center,  Pacific,  conducted  further  tests  at  their  Integration 
Test  Facility  using  archived  and  compressed  SPS-48E/HWDDC  UF  files.  These  tests 
measured  the  bandwidth  load  during  the  transmission  of  a  series  of  compressed  UF  files 
from  a  HWDDC  to  a  simulated  FNMOC  Finux  Server.  The  simulation  was  conducted 
continuously  over  a  24  hour  period  and  successfully  met  the  bandwidth  requirements. 
Having  passed  all  the  required  tests,  the  HWDDC  has  since  been  installed  on  the  USS 
Boxer  (FHD-4)  and  the  USS  Reagan  (CVN-76),  and  ten  more  ships  are  scheduled  for 
installations  by  October  2012. 


TRANSITIONS 

As  a  result  of  the  collaboration  with  NRF  researchers,  UFZIP/  UFUNZIP  (Version  1.0): 

UF  files  Compression  and  Decompression  Software  Package,  was  transitioned  to 

SPAWAR/HWDDC,  on  19  June  2009. 
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